Skip to content

Fix concurrent Lambda invocation trace isolation - #761

Merged
jj22ee merged 3 commits into
aws:masterfrom
maharshi-coding:fix/lambda-invocation-trace-isolation
Nov 11, 2026
Merged

jj22ee merged 3 commits into
aws:masterfrom
maharshi-coding:fix/lambda-invocation-trace-isolation

Conversation

@maharshi-coding

@maharshi-coding maharshi-coding commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #760.

Concurrent Lambda Managed Instance invocations can share an active CLS segment and facade. This can attach annotations to another invocation, emit a subsegment on the wrong trace, or drop sampled work after an unsampled invocation resolves the shared facade.

Use the runtime's InvokeStore context identity to give each invocation its own facade and CLS segment slots. The slots follow the CLS prototype chain so nested and parallel capture scopes restore their caller. Weak keys let expired contexts and tracing state be collected. Startup initializes the legacy CLS slot, and runtimes without an invocation context use that fallback.

The context helpers cache Lambda detection at module load and load the Lambda module on first use. An attempt to set a segment without an active CLS context logs the existing warning and preserves the invocation facade.

The isolation suite uses the runtime's InvokeStore 0.2.2. It covers concurrent invocations and classic Lambda, whose single-context store retains the previous context between handlers. The classic test leaves a subsegment assigned in one handler and verifies that the next gets a fresh facade and emits on its own trace. The missing-context test verifies the warning and exercises capture in a new CLS scope.

Local validation

  • Clean npm 8.19.4 install, hoisted bootstrap, build, all 11 package test targets, and all 10 lint targets passed on Node 14.21.3 in x86 Linux Docker and Node 16.20.2 on macOS. The packaged SDK smoke test passed on Node 16.
  • All 11 Lambda isolation tests passed on Node 14, 16, 20, and 22. The classic test fails against upstream 458fc17 and the missing-context test fails with the previous invocation fallback. Local coverage hit all 37 executable lines changed by this PR, including the warning.
  • In ARM Linux Docker, the Node 14 full core suite crashes at the same existing Lambda unit test boundary on the previous PR head and the updated code. The isolated Lambda unit and isolation suites pass.

The reporter's unchanged handler reproduced cross-attribution before the initial isolation patch. With that patch, the local replay emitted 30 correctly attributed documents in each of the three modes. The initial patch's separate 10,000-invocation stress run observed zero attribution errors and emitted the expected 13,334 sampled documents.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@maharshi-coding
maharshi-coding requested a review from a team as a code owner November 2, 2026 02:40
All Build and Lint jobs fail at the install step because the committed
lockfile pins semver@6.3.1, while the ^6.3.0 range now resolves to
semver@6.3.2 (published upstream since the lockfile was last updated):

  Invalid: lock file's semver@6.3.1 does not satisfy semver@6.3.2

Regenerate the three semver entries to 6.3.2 so package.json and
package-lock.json are back in sync. No other dependencies change and
lockfileVersion stays 2.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.29630% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 85.24%. Comparing base (458fc17) to head (e7f05f6).

Files with missing lines Patch % Lines
packages/core/lib/context_utils.js 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #761      +/-   ##
==========================================
+ Coverage   84.31%   85.24%   +0.92%     
==========================================
  Files          36       36              
  Lines        1817     1843      +26     
==========================================
+ Hits         1532     1571      +39     
+ Misses        285      272      -13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jj22ee jj22ee self-assigned this Nov 7, 2026
Comment thread packages/core/lib/context_utils.js Outdated
if (cls_mode) {
var segment = contextUtils.getNamespace(NAMESPACE).get(SEGMENT);
var namespace = contextUtils.getNamespace(NAMESPACE);
var invocation = process.env.LAMBDA_TASK_ROOT && require('./env/aws_lambda').getInvocationContext();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reads process.env.LAMBDA_TASK_ROOT on every getSegment() call, and setSegment() does the same on line 124. Both are on the hot path and process.env lookups aren't free, so non-Lambda users pay for this too. Could you resolve it once at module load (for example a module-level const isLambda = !!process.env.LAMBDA_TASK_ROOT) and reuse it in both places?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

require('./env/aws_lambda') also runs on every getSegment()/setSegment() call, including every .then() when capturePromise() is enabled. Can you also cache the module lazily on first use?

Comment thread packages/core/lib/env/aws_lambda.js Outdated
if (active) {
segments.set(active, current);
} else {
segment = current;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When there's no active CLS context, this replaces the invocation's fallback segment, so every later getSegment() in this invocation without a mapped slot returns current instead of the facade. Before this change, setSegment() without an active context failed and logged "Failed to set the current sub/segment on the context." without changing anything. Unless there's a case that needs this, could it keep the old behavior, e.g. return false here so contextUtils.setSegment() logs the same warning, instead of silently swapping the invocation's root?

@jj22ee jj22ee left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ty for this fix! added a couple small requests.

Comment thread packages/core/lib/context_utils.js Outdated
if (cls_mode) {
var segment = contextUtils.getNamespace(NAMESPACE).get(SEGMENT);
var namespace = contextUtils.getNamespace(NAMESPACE);
var invocation = process.env.LAMBDA_TASK_ROOT && require('./env/aws_lambda').getInvocationContext();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

require('./env/aws_lambda') also runs on every getSegment()/setSegment() call, including every .then() when capturePromise() is enabled. Can you also cache the module lazily on first use?

Cache the Lambda runtime check and load the context helper lazily. Preserve the invocation facade and log the existing warning when no CLS context is active.

Test the classic runtime's persistent InvokeStore and isolate Lambda tests so the runtime flags are set before the SDK loads.

Assisted-by: OpenAI Codex
@jj22ee
jj22ee merged commit fca7825 into aws:master Nov 11, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lambda mode shares one CLS context and facade segment across concurrent invocations (Lambda Managed Instances)

4 participants